Papers with multimodal systems
Understanding the Effect of Textual Adversaries in Multimodal Machine Translation (D19-64)
Copied to clipboard
| Challenge: | Existing studies show that multimodal machine translation systems are better than text-only systems at translating phrases that have a direct correspondence in the image. |
| Approach: | They conduct experiments with both visual and textual adversaries to understand the role of textual inputs in multimodal machine translation. |
| Outcome: | The proposed model can recover masked tokens in the source sentences . the proposed model is based on a model with a visual modality . |
“I’ve Seen Things You People Wouldn’t Believe”: Hallucinating Entities in GuessWhat?! (2021.acl-srw)
Copied to clipboard
| Challenge: | a problem with natural language generation systems is the generation of tokens that are unrelated to the source input. |
| Approach: | They propose two new models to play the GuessWhat?! referential game . they propose to adapt the best visual processing models available to mitigate this issue . |
| Outcome: | The proposed models generate few hallucinations compared to other models available in the literature. |
Improving Multimodal fusion via Mutual Dependency Maximisation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Multimodal sentiment analysis is a trending area of research, and multimodal fusion is one of its most active topics. |
| Approach: | They propose to use modality-based penalties to measure dependency between models to improve accuracy. |
| Outcome: | The proposed methods improve accuracy on two well-known sentiment analysis datasets by 4.3 on the proposed models and by-product includes a statistical network which can interpret the high dimensional representations learnt by the model. |
Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework (2026.eacl-srw)
Copied to clipboard
Grzegorz Statkiewicz, Alicja Dobrzeniecka, Karolina Seweryn, Aleksandra Krasnodębska, Karolina Piosek, Katarzyna Bogusz, Sebastian Cygert, Wojciech Kusa
| Challenge: | Currently, most vision-language models are trained on English-centric data, limiting their usability for non-English-speaking users. |
| Approach: | They reproduce and adapt LLaVA-Next methodology to create Polish VLMs . they use a fully automated pipeline for translating and filtering existing multimodal datasets based on Polish data for OCR and culturally specific tasks. |
| Outcome: | The proposed model improves on a Polish-adapted model and shows higher quality captions in generative evaluations. |
SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs (2025.emnlp-main)
Copied to clipboard
Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Yuanxing Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Wentao Zhang, Feng Zhao
| Challenge: | Multimodal Large Language Models (MLLMs) integrate visual and textual inputs, yet modality alignment remains one of the most challenging aspects. |
| Approach: | They propose a token-level supervision alignment method that enables more precise visual-text alignment during pretraining. |
| Outcome: | The proposed method improves performance across various model sizes, with smaller models benefiting the most. |
MIRAGE: Metadata-guided Image Retrieval and Answer Generation for E-commerce Troubleshooting (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing multimodal systems often associate text and images based on embedding similarity or simple co-location, but fail to ensure that the linked image accurately depicts the specific product or component mentioned in a troubleshooting instruction. |
| Approach: | They propose a metadata-first paradigm that treats structured metadata as a modality for multimodal grounding. |
| Outcome: | The proposed model uses a semantic schema to capture product attributes and visual aspects. |
How Do Inpainting Artifacts Propagate to Language? (2026.acl-short)
Copied to clipboard
| Challenge: | Figure 1 shows representative examples of visual artifacts introduced by diffusion-based inpainting . despite visually plausible reconstructions, localized inpainding artifactors lead to object substitutions, attribute changes, or category-level errors in downstream captions. |
| Approach: | They propose a diagnostic setup in which masked image regions are reconstructed and then provided to captioning models. |
| Outcome: | The proposed diagnostic framework can be used to examine how visual artifacts affect language generation in vision-language models. |
MultiNERD: A Multilingual, Multi-Genre and Fine-Grained Dataset for Named Entity Recognition (and Disambiguation) (2022.findings-naacl)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a process of identifying named entities in unstructured texts and classifying them through specific semantic categories. |
| Approach: | They propose a method for automatically producing NER annotations and introduce a manually-annotated test set. |
| Outcome: | The proposed method covers 10 languages, 15 NER categories and 2 textual genres and a manually-annotated test set. |
Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to quantify uncertainty are limited in vision-language models . however, current models display notable miscalibration across diverse tasks and settings . |
| Approach: | They evaluate verbalized confidence in vision-language models using visual reasoning . they propose a prompting strategy that improves confidence alignment in multimodal settings . |
| Outcome: | The proposed method improves confidence alignment across multimodal settings. |
Simultaneous Machine Translation with Visual Context (2020.emnlp-main)
Copied to clipboard
| Challenge: | Simultaneous machine translation (SiMT) aims to reproduce human interpretation, where an interpreter translates spoken utterances as they are produced. |
| Approach: | They propose to add visual context to siMT to compensate for the missing source context . they show visual-grounded models are much better than commonly used global features . |
| Outcome: | The proposed models reach up to 3 BLEU points improvement under low latency scenarios. |
Towards Speech-only Opinion-level Sentiment Analysis (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing systems that estimate user preferences only in static manners or exploit interaction history are inadequate to accurately assess user preferences. |
| Approach: | They propose to integrate rank consistent ordinal regression into a speech-only sentiment prediction task performed by ResNet-like systems and use speaker verification extractors trained on larger datasets as low-level feature extractor. |
| Outcome: | The proposed system beats state-of-the-art unimodal systems on multimodal Opinion Sentiment and Emotion Intensity databases. |
The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent advances in large vision-language models have led to remarkable progress in complex visual understanding across scientific and reasoning tasks. |
| Approach: | They evaluate 18 state-of-the-art vision-language models across 6 multimodal datasets with 3 distinct scoring functions and develop instruction-guided likelihood proxies for closed-source models lacking token-level logprob access. |
| Outcome: | The proposed model is able to achieve higher accuracy on multimodal benchmarks while performing poorer on reasoning tasks. |
Unveiling Multimodal Processing: Exploring Activation Patterns in Multimodal LLMs for Interpretability and Efficiency (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have remained opaque. |
| Approach: | They propose a method to convert dense MLLMs into fine-grained Mixture-of-Experts architectures. |
| Outcome: | The proposed method outperforms random expert pruning and sparse activation and model pruning. |
K-GIP: Diagnosing Logical Fractures in Large Vision-Language Models via Verification Scene Graphs and Sequential Pruning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks that treat hallucinations as isolated errors neglect causal dependencies between visual perception and textual reasoning. |
| Approach: | They propose a Knowledge-Guided In-Context Probing framework that constructs a dual-perception ground truth to transform abstract priors into multi-granularity queries. |
| Outcome: | The proposed framework isolates deep reasoning failures from simple perceptual misses. |
Multimodality for NLP-Centered Applications: Resources, Advances and Frontiers (2022.lrec-1)
Copied to clipboard
| Challenge: | resurgence of multimodal datasets has attracted significant research interest, but there is no comprehensive survey for this task. |
| Approach: | They present a survey of a multimodal dataset with different modalities according to the applications. |
| Outcome: | The proposed datasets are available online and discuss the new frontier and motivate future researches. |
Misalignment Attack on Text-to-Image Models via Text Embedding Optimization and Inversion (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Text embedding is a key component of modern NLP models but also poses additional risks. |
| Approach: | They propose a framework that optimizes embeddings and inverts them to obtain misaligned prompts. |
| Outcome: | The proposed framework exploits the continuity and distribution characteristics of text embeddings to obtain misaligned prompts of discrete tokens. |
SoundBreak: A Systematic Study of Audio-Only Adversarial Attacks on Trimodal Models (2026.acl-long)
Copied to clipboard
| Challenge: | Recent advances in multimodal large language models have increased their vulnerability to adversarial manipulation. |
| Approach: | They propose to target audio-only adversarial attacks on multimodal audio–video–language models . they show that attacks can be successful at low perceptual distortions . |
| Outcome: | The proposed models achieve up to 96% success rate under realistic conditions . the proposed models are more robust to noise than to noise and distortion than to speech recognition systems . |
Beyond Content: How Grammatical Gender Shapes Visual Representation in Text-to-Image Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | grammatical gender significantly influences image generation in text-to-image models . masculine grammatikal markers increase male representation to 73% on average . feminine grammatological markers increase female representation to 38% . |
| Approach: | They propose a cross-linguistic benchmark examining words where grammatical gender contradicts stereotypical gender associations. |
| Outcome: | The proposed benchmark examines words where grammatical gender contradicts stereotypical gender associations. |
Concept-pedia: a Wide-coverage Semantically-annotated Multimodal Dataset (2025.emnlp-main)
Copied to clipboard
| Challenge: | Current evaluations for Vision-language Models remain heavily anchored to ImageNet . |
| Approach: | They propose a large-scale semantically-annotated multimodal resource that extends the range of visual concepts, including diverse abstract categories. |
| Outcome: | The proposed model expands the range of visual concepts, including diverse abstract categories. |